Device and method for panoramic image processing

ABSTRACT

A device includes one or more image sensors that capture live video data and a memory that stores the captured live video data. The device includes circuitry that determines an angle-of-view for one or more frames of the live video data and generates wide angle video data that has a larger angle-of-view than the one or more frames of the live video data. The circuitry determines one or more angles-of-view not represented in the wide angle video data by the live video data. The circuitry generates make-up image data captured at a later time than the live video data, wherein the make-up image data has an angle-of-view corresponding to at least one of the one or more angles-of-view that are not represented in the wide angle video data by the live video data. The circuitry updates the wide angle video data to include the make-up image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation application of and claims the benefit of priorityunder 35 U.S.C. §120 from U.S. application Ser. No. 15/048,730, filedFeb. 19, 2016, which is a continuation of U.S. application Ser. No.14/192,446, filed Feb. 27, 2014, now U.S. Pat. No. 9,300,882, the entirecontents of each of which is incorporated herein by reference.

BACKGROUND

Technical Field

The present disclosure relates to image processing and, in particular,the processing of video signals.

Description of Related Art

Terminal devices such as smartphones include cameras that can performrecording of moving images and/or still images. When recording a movingimage or still image, the lens of the camera mounted on the terminaldevice captures an area in front of the lens with a finiteangle-of-view. There exist some cameras that can capture images with awide angle-of-view. For example, there exists a camera that can capture360-degrees of the surrounding area with respect to the camera.Moreover, there exist cameras with installed video processing programsthat can capture a plurality of images in an area surrounding the cameraand connect the plurality of images such that a 360-degree panoramicimage is obtained. For example, a user may have a terminal device with acamera that performs recording with a 45-degree angle-of-view. In thisexample, the user may capture eight images, whereby each of the imagesis captured after rotating the camera approximately 45-degrees such thata combination of the eight images forms a 360-degree panoramic imagewhen the images are connected.

However, with respect to moving image data and related video signals,even if image data is captured by changing a direction of a camera, a360-degree panoramic image (or size image having an angle-of-viewgreater than the recording angle-of-view of the camera) cannot beobtained with the moving image data. That is, in the case in which avideo signal is captured by rotating a camera, the timings at which thelive image data is obtained for each angle at which the camera isrotated will differ and therefore, the images cannot be combined to forma single panoramic image.

To illustrate the above problem by way of example, FIG. 15 illustrates apanoramic image P0, which represents a 360-degree angle-of-view. In thisexample, angle-of-view F0 represents the angle-of-view in which a cameraused in the recording can actually capture an image. If video recordingis performed while rotating the camera such that the angle-of-viewchanges from angle-of-view F1 to angle-of-view F2 to angle-of-view F3, apanoramic video image cannot be formed using the images captured withrespect to each angle-of-view (as shown in panoramic images P1, P2, andP3) because live image data does not exist in the surrounding area withrespect to each of the angle-of-views. In other words, because liveimage data can only be captured with respect to the angle-of-view inwhich the camera is currently facing, panoramic videos cannot be formedfor a given time instant by combining a combination of moving image datain the limited angle-of-views because live image data does not exist atthe different timings at which the recording was performed.

SUMMARY

In one or more embodiments according to the present disclosure, a deviceincludes one or more image sensors that capture live video data and amemory that stores the captured live video data. The device includescircuitry configured to determine an angle-of-view for one or moreframes of the live video data and to generate wide angle video data thathas a larger angle-of-view than the one or more frames of the live videodata. The circuitry is configured to determine one or moreangles-of-view not represented in the wide angle video data by the livevideo data. The circuitry is configured to generate make-up image datacaptured at a later time than the live video data, wherein the make-upimage data has an angle-of-view corresponding to at least one of the oneor more angles-of-view that are not represented in the wide angle videodata by the live video data. The circuitry is configured to update thewide angle video data to include the make-up image data.

The foregoing general description of the illustrative embodiments andthe following detailed description thereof are merely exemplary aspectsof the teachings of this disclosure, and are not restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of this disclosure and many of theattendant advantages thereof will be readily obtained as the samebecomes better understood by reference to the following detaileddescription when considered in connection with the accompanyingdrawings, wherein:

FIGS. 1A-1C illustrate an outline of video processing performed in oneor more embodiments in accordance with the present disclosure;

FIG. 2 illustrates a block diagram for an exemplary terminal device,according to certain embodiments;

FIG. 3 illustrates an exemplary flow chart for generating and storingimage data, according to certain embodiments;

FIG. 4 illustrates a non-limiting exemplary flow chart of processingrelated to the reproduction of video data stored in a memory, accordingto certain embodiments;

FIG. 5 illustrates a non-limiting exemplary flow chart for generatingwide angle video data, according to certain embodiments;

FIG. 6 illustrates an exemplary sequential flow diagram at a time ofrecording video data, according to certain embodiments;

FIG. 7 illustrates an exemplary flow diagram of image data processing ata time of video reproduction, according to certain embodiments;

FIGS. 8A through 8C illustrate exemplary reproduction states ofpanoramic image data, according to certain embodiments;

FIGS. 9A and 9B illustrate exemplary relationships between recordingangle-of-view of a captured image and reproduction angle-of-view of adisplayed image, according to certain embodiments;

FIGS. 10A and 10B illustrate an example of simulating motion of amake-up image displayed within wide angle video data, according tocertain embodiments;

FIGS. 11A through 11C illustrate another non-limiting example ofsimulating motion of a make-up image within wide angle video data,according to certain embodiments;

FIG. 12 illustrates a non-limiting exemplary sequential flow diagram forsimulating motion of a make-up image, according to certain embodiments;

FIG. 13 illustrates a non-limiting exemplary sequential flow diagram forreproducing images with simulated motion of a make-up image, accordingto certain embodiments;

FIG. 14 illustrates a non-limiting example of capturing image data withcameras mounted on multiple sides of a terminal device, according tocertain embodiments; and

FIG. 15 illustrates exemplary panoramic image data.

DETAILED DESCRIPTION

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts throughout the several views.

FIGS. 1A-1C illustrate an outline of video processing performed in oneor more embodiments in accordance with the present disclosure.

Referring first to FIG. 1A, FIG. 1A illustrates multiple angles-of-viewin which image data is captured with respect to a panoramic image P10.In this example, the panoramic image P10 includes image data fromangles-of-view F11-F15. That is, the angle-of-view at which a camera inthis example captures video data may be any one of angle-of-viewsF11-F15. In addition, the darkened area in the panoramic image P10represents areas of the panoramic image P10 in which image data is notcaptured using the camera.

FIG. 1B illustrates exemplary processing performed when changing anangle-of-view of a camera while capturing video data. Image datacaptured at a time at which the camera records the video image data isreferred to herein as live image data. In this example, it is assumedthat a camera initially captures live image data at an angle-of-viewcorresponding to F 11 from FIG. 1A. The camera then pans to the right tocapture live image data at the angle-of-view F12, and then the camerapans back to the left to capture live image data at angle-of-view F13.As shown in FIG. 1B, live image data is captured with respect to thefront, right, and left sides of the camera as the camera is rotatedwhile recording the video as described above. Accordingly, theangle-of-view in FIG. 1B at which live image data may be included inwide angle video data varies in accordance with the movement of thecamera over time. Hereinafter, the term wide angle video data (or wideangle image data) may be utilized interchangeably with the termpanoramic image/video data to describe image data having a relativelywide angle-of-view.

At a time during or after the capturing of the video data in the aboveexample, the video data may be reproduced on a display included on aterminal device. For example, as shown in FIG. 1C, video data may bereproduced on a video reproduction apparatus 200, which is representedin this example as wearable glasses. It may be assumed that the videoreproduction apparatus 200 includes various sensors for detecting amovement and determining an orientation of the video reproductionapparatus 200 (i.e. a movement of the user's head while wearing theglasses). For example, as shown in FIG. 1C, the sensors included in thevideo reproduction apparatus 200 may detect that an orientation of theuser's head corresponding to a center angle-of-view of the panoramicimage P10. The orientation of the video reproduction apparatus 200 mayrelate to the display angle-of-view when image data is reproduced on theapparatus.

As motion sensors included in the video reproduction apparatus 200detect changes in the orientation of the video reproduction apparatus200, control circuitry (e.g., a processor or microcontroller device)included in the apparatus may control the display outputting thepanoramic image P10 such that the angle-of-view seen by the userchanges. Here, if it is assumed that image data exists for a range P10′within the panoramic image P10, as the orientation of the videoreproduction apparatus 200 changes, the displayed angle-of-viewcorrespondingly changes to output image data within the range P10′. Forexample, when the user tilts his or her head upwards, image data abovethe center region Fa is reproduced in the video reproduction apparatus200. Similarly, if the user moves his or her head left, right, ordownwards, image data is reproduced in the video reproduction apparatuswith a corresponding angle-of-view of the direction of motion of theuser's head with respect to the center region Fa and the range P10′.

However, as discussed previously, live image data for a video may onlybe captured for a particular angle-of-view for any given time instant.Therefore, although the user may move his or her head such that theorientation of the video reproduction apparatus 200 changes, andalthough the output in the display may change angle-of-view inaccordance with the orientation of the user's head, there may notnecessarily exist live image data corresponding to the angle-of-viewcurrently displayed in the video reproduction apparatus 200.

For this reason, live image data captured at a different time than thetime corresponding to the current display time within the video may beutilized when reproducing wide angle video data at a given angle-of-viewfor the panoramic image P10. For example, when the user's head facesleftward during video reproduction such that the orientation of thevideo reproduction apparatus 200 changes to the left with respect to thecenter region Fa, the video reproduction apparatus 200 may reproducevideo image data with an angle-of-view corresponding to angle-of-viewF13. However, there may not exist live image data corresponding to thetime at which the orientation of the video reproduction apparatus 200changes to the left when reproducing the image data corresponding to theangle-of-view F13. Therefore, the image data used for reproducing theimages in the angle-of-view F13 at this time may be set as make-up imagedata generated from live image data recorded at the angle-of-view F13 ata later time, as shown in FIG. 1B.

Referring to FIG. 1B, assuming the polygon shapes in the figurerepresent sequential video frames that may be included in wide anglevideo data, if a request was initially received to reproduce image datacorresponding to the angle-of-view F13, control circuitry in the videoreproduction apparatus 200 may determine that live image data does notexist in the angle-of-view F13 at the time at which reproduction of theimage data is requested (i.e. live image data is initially captured fromangle-of-view F11 in the example of FIG. 1B). However, the controlcircuitry of the video reproduction apparatus 200 may determine thatlive image data was captured for the angle-of-view F13 at a later time.Based on this determination, make-up image data may be generated for theangle-of-view F13 using the live image data captured for theangle-of-view F13 as the later time, and the make-up image data for theangle-of-view F13 may be reproduced in a wide angle video at a timecorresponding to the time at which the live image data for angle-of-viewF 11 was captured. Similar processing may also be performed forgenerating make-up image data corresponding to the angle-of-view F12 (oranother arbitrary angle-of-view). Accordingly, wide angle video data maybe reproduced based on a combination of live image data and make-upimage data corresponding to the various angle-of-views captured by thecamera.

In one or more embodiments, the make-up image data in the generated wideangle video data may be displayed during reproduction until a timecorresponding to the time when the make-up image data was captured live.That is, when make-up image data is displayed in wide angle video datain lieu of live image data (e.g., because live image data is notrepresented for an angle-of-view requested when displaying the wideangle video data), the video reproduction apparatus 200 may be in astate in which still image data corresponding to the make-up image datais displayed (e.g., the “no update” state shown in FIG. 1B).

Next, FIG. 2 illustrates a block diagram for an exemplary terminaldevice, according to certain embodiments of the present disclosure. Forsimplicity, the terminal device illustrated in FIG. 2 is implemented asa mobile phone terminal device in examples described herein. However,the skilled artisan will appreciate that the processing described hereinmay be easily adapted for other types of devices (e.g. a desktopcomputer, a tablet device, a laptop computer, a server, an e-readerdevice, a camera, a navigation unit, etc.).

Referring now to FIG. 2, the exemplary terminal device 100 of FIG. 2includes a controller 110, a wireless communication processor 102connected to an antenna 101, a voice processor 103, a speaker 104, amicrophone 105, a short distance wireless communication processor 107connected to an antenna 106, a motion sensor 108, a camera 109, adisplay 120, a touch panel 130, an operation key 140, and a memory 150.

The antenna 101 transmits/receives electromagnetic wave signals betweenbase stations for performing radio-based communication, such as thevarious forms of cellular telephone communication.

The wireless communication processor 102 controls communicationsperformed between the terminal device 100 and other external devices viathe antenna 101. For example, the wireless communication processor 102may control communication between base stations for cellular telephonecommunication.

The voice processor 103 demodulates and/or decodes the audio data readfrom the memory 150, or audio data received by the wirelesscommunication processor 102 and/or short-distance wireless communicationprocessor 107. Additionally, the voice processor 103 may decode audiosignals received from the microphone 105.

The speaker 104 emits an audio signal corresponding to audio datasupplied from the voice processor 103.

The microphone 105 detects surrounding audio, and converts the detectedaudio into an audio signal. The audio signal may then be output to thevoice processor 103 and/or the controller 110 for further processing.

The antenna 106 may transmit/receive electromagnetic wave signalsto/from other external apparatuses, and the short-distance wirelesscommunication processor 107 may control the wireless communicationperformed with the other external apparatuses. Bluetooth, IEEE 802.11,and near field communication (NFC) are non-limiting examples of wirelesscommunication protocols that may be used for inter-device communicationvia the short-distance wireless communication processor 107.

The sensor 108 may include one or more motion sensors capable ofdetermining various aspects of motion with respect to the terminaldevice 100. For example, the sensor 108 may include one or moregyroscopes, accelerometers, or the like. In certain embodiments, thecontroller 110 may determine an orientation of the terminal device 100based on motion sensor inputs received from the motion sensor 108. Theorientation of the terminal device 100 may be represented in one or moreembodiments as a tilt angle of the terminal device 100 with respect toone or more axes (e.g., the x/y/z axis). In one or more embodiments, theorientation of the terminal device 100 may include an indication ofwhether the terminal device 100 is held in a landscape or portraitorientation. In one or more embodiments, the orientation of the terminaldevice 100 may correspond to an angle-of-view at a time of recordingand/or reproducing image data with the terminal device 100.

Image data may be generated by the terminal device 100 via the camera109, which may include one or more image sensors comprised of, e.g., acharged couple device (CCD), complementary metal oxide semiconductor(CMOS), or the like. For example, an image signal may be generated bythe camera 109 when an image formed on a light-receiving surface througha lens is photoelectrically converted. The lens of the camera 109 may,for example, be arranged on a front end or back surface of the terminaldevice 100 when implemented as a mobile phone device or smartphone. Thecamera 109 may, in one or more embodiments, include one or moreprocessing circuits for performing processing features with respect tostill and/or moving image data. Additionally, the controller 110 mayexecute panoramic image processing features described herein withrespect to the still and/or moving image data generated by the camera109.

The controller 110 may include one or more central processing units(CPUs), and may control each element in the terminal device 100 toperform features related to communication control, audio signalprocessing, control for the audio signal processing, image processingand control, and other kinds of signal processing. The controller 110may perform these features by executing instructions stored in thememory 150. Alternatively or in addition to the local storage of thememory 150, the features may be executed using instructions stored in anexternal device accessed on a network or on a non-transitory computerreadable medium.

The display 120 may be a liquid crystal display (LCD), an organicelectroluminescence display panel, or another display screen technology.In addition to displaying still and moving image data, the display 120may display operational inputs, such as numbers or icons, which may beused for control of the terminal device 100. The display 120 mayadditionally display a graphical user interface with which a user maycontrol aspects of the terminal device 100. Further, the display 120 maydisplay characters and images received by the terminal device 100 and/orstored in the memory 150 or accessed from an external device on anetwork. For example, the terminal device 100 may access a network suchas the Internet and display text and/or images transmitted from a Webserver.

The touch panel 130 may include a physical touch panel display screenand a touch panel driver. The touch panel 130 may include one or moretouch sensors for detecting an input operation on an operation surfaceof the touch panel display screen. Used herein, the phrasing “touchoperation” refers to an input operation performed by touching anoperation surface of the touch panel display with an instruction object,such as a finger or stylus-type instrument. In the case where a stylus,or the like, is used in a touch operation, the stylus may include aconductive material at least at the tip of the stylus such that thesensors included in the touch panel 130 may detect when the stylusapproaches/contacts the operation surface of the touch panel display(similar to the case in which a finger is used for the touch operation).

In one or more embodiments, the touch panel 130 may be disposed adjacentto the display 120 (e.g., laminated), or may be formed integrally withthe display 120. The display 120 and the touch panel 130 may besurrounded by a protective casing, which may also enclose the otherelements included in the terminal device 100.

In one or more embodiments, the touch panel 130 is a capacitance-typetouch panel technology. In other embodiments, the touch panel 130 may beimplemented using other touch panel types with alternative structures,such as resistance-type touch panels. In certain embodiments, the touchpanel 130 may include transparent electrode touch sensors arranged inthe x/y direction on the surface of transparent sensor glass.

The touch panel driver may be included in the touch panel 130 forcontrol processing related to the touch panel 130, such as scanningcontrol. For example, the touch panel driver may scan each sensor in anelectrostatic capacitance transparent electrode pattern in the xdirection and the y direction, and detect the electrostatic capacitancevalue of each sensor to determine when a touch operation is performed.The touch panel driver may output a coordinate and correspondingelectrostatic capacitance value for each sensor. The touch panel drivermay also output a sensor identifier that may be mapped to a coordinateon the touch panel display screen. Additionally, the touch panel driverand touch panel sensors may detect when an instruction object is withina predetermined distance from the operation surface of the touch paneldisplay screen.

The operation key 140 may include one or more buttons similar toexternal control elements. The operation key 140 may generate anoperation signal based on a detected input. The operation signalsgenerated by the operation key 140 may be supplied to the controller 110for performing related processing control of the terminal device 100. Incertain aspects of the present disclosure, the processing and/orfunctions associated with external buttons and the like may be performedby the controller 110 in response to an input operation on the touchpanel 130 in lieu of implementing the terminal device with externalbuttons in the operation key 140.

The terminal device 100 includes a control line CL and a data line DL asinternal communication bus lines. Control data to/from the controller110 may be transmitted through the control line CL. The data line DL maybe used for transmission of voice data, display data, etc.

Next, FIG. 3 illustrates an exemplary flow chart for generating andstoring image data, in accordance with certain embodiments of thepresent disclosure. In one or more embodiments, the processingillustrated in FIG. 3 may be performed by the controller 110 and thecamera 109.

Referring to FIG. 3, the process begins at step S21 where the camera 109captures image data and stores the captured image data in the memory150. In this example, it is assumed that one frame of image data iscaptured at step S21. Accordingly, when video data is captured at stepS21, a series of frames may be captured and stored by repeating steps ofFIG. 3 such that video data may be reproduced from the sequential framescorresponding to the captured image data.

In one or more embodiments, when capturing the image data with thecamera 109, the camera 109 may determine an angle-of-view and a framenumber corresponding to the frame of the image data captured by thecamera. The angle-of-view and the frame number corresponding to theframe of the captured image data may be stored in the memory 150 at stepS22. A frame number may be represented as a consecutive number beginningat a frame captured the start of recording and ending at a framecaptured at the finish of recording. When capturing video data, theframe number may sequentially increase from the initial frame capturedto the last frame captured in the video data. In one or moreembodiments, the angle-of-view stored at step S22 may be generated bythe sensor 108. That is, the angle-of-view stored at S22 may begenerated based on a determined orientation of the terminal device 100when the image data is captured.

At step S23, the controller 110 determines if the recording of imagedata has been terminated. For example, the controller 110 may determineat step S23 that the capturing of video data by the camera 109 has beencompleted based on an input received from the operation key 140 or atouch operation received on the touch panel 130. If recording has ended,the process of FIG. 3 is complete. Otherwise, the process returns tostep S21 and additional frames of image data are captured to generate avideo.

Next, FIG. 4 illustrates a non-limiting exemplary flow chart ofprocessing related to the reproduction of video data stored in a memory,according to certain embodiments. In this example, the display 120 ofthe terminal device 100 may be utilized for reproducing the video datastored in the memory 150. Further, processing related to generating wideangle video data based on the live image data captured by the camera 109may be performed under the control of the controller 110. Moreover,while the present example describes processing related to reproducingvideo data at a time that is subsequent to the time at which the imagedata is recorded, the processing in FIG. 4 may be adapted such that itis performed at other times, such as at a time corresponding to therecording of the image data.

At step S11, the controller 110 starts deploying images of allangles-of-view for generating and reproducing wide angle video data.Exemplary processing related to generating wide angle video data will bedescribed in detail later at least with respect to FIG. 4.

Next, the controller 110 at step S12 determines an orientation of theterminal device 100 based on inputs received from the sensor 108. In oneor more embodiments, the orientation determined at step S12 includes aviewing angle of the terminal device 100.

At step S13, the controller 110 reads out image data stored in thememory 150 that corresponds to the orientation determined at step S12.In one or more embodiments, the image data read out by the controller110 at step S13 has a corresponding angle-of-view to the orientationdetermined at step S12.

At step S14, the controller 110 determines if an input is receivedindicating a termination of video reproduction. If an input is receivedindicating the termination of video data reproduction, the processing ofFIG. 4 ends at this point. Otherwise, the controller 110 returns to stepS12.

Next, FIG. 5 illustrates a non-limiting exemplary flow chart forgenerating wide angle video data, according to certain embodiments. Theexemplary processing illustrated in FIG. 5 represents a process forgenerating wide angle video data based on a combination of live imagedata and make-up image data for a single frame. Therefore, thecontroller 110 may perform the process illustrated in FIG. 5 for aseries of frames captured in a given time interval. In other words, theprocessing of FIG. 5 may be repeated for a series of frames such thatwide angle video data for the series of frames is generated based on thecombination of live image data and make-up image data.

At step S31, the controller 110 sets an angle-of-view for generating thewide angle video data. For example, assuming that the wide angle videodata has a greater angle-of-view than one or more frames of image dataused to generate the wide angle video data, the controller 110 at stepS31 determines a portion of the greater angle-of-view in which to beginthe processing of expanding the captured image to form the wide anglevideo data.

At step S32, the controller 110 determines whether live image dataexists for the angle-of-view set at step S31. That is, the controller110 determines whether live image data stored in the memory 150 has acorresponding angle-of-view to the angle-of-view set at step S31. If thecontroller 110 determines that live image data exists for theangle-of-view set at step S31, the controller 110 at step S33 sets thestored live image data as the image data utilized for generating thewide angle video data for the angle-of-view set at step S31. In otherwords, when the wide angle video data is reproduced when theangle-of-view set at step S31 is visible at a time corresponding to thepresent frame, live image data will be reproduced in an area of the wideangle video data corresponding to the angle-of-view set at step S31.

At step S34, if the controller 110 at step S32 determines that liveimage data does not exist for the time period corresponding to thepresent frame, the controller 110 at step S34 determines whether thereis image data corresponding to the angle-of-view set at step S31 thatwas recorded at a time later than the time corresponding to the presentframe. In other words, if the camera 109 is directed to an angle-of-viewthat does not correspond to the angle-of-view at step S31 andconsequently live image data is recorded for an image data that does notcorrespond to the angle-of-view set at step S31, the controller 110 atstep S34 determines whether the angle-of-view set at step S31 wasrecorded at a time later than the time at which the live image data forthe present frame was recorded. That is, the controller 110 determineswhether make-up image data may be generated for the angle of view set instep S31.

For example, referring to FIG. 1B, assume that the angle-of-view set atstep S31 of FIG. 5 is angle-of-view F12 corresponding to the right sideof the panoramic image P10. As illustrated in FIG. 1B, live image datafor the first frame in a series of frames of image data represented inFIG. 1B exists only for the angle-of-view F11. Therefore, the controller110 generates image data for the angle-of-view not represented by liveimage data by utilizing make-up image data that was recorded at a timesubsequent to the time that the live image data for the present frame(i.e., the first frame in FIG. 1B) was recorded. As a result, wide anglevideo data may be generated in the first frame in the series of framesof FIG. 1B by combining the live image data from angle-of-view F11 withthe makeup data from angles-of-view F12 and F13.

Referring back to FIG. 5, if the controller 110 determines at step S34that live image data was not recorded at a time later than the presentframe for the angle-of-view set at step S31, the controller 110 at stepS36 generates the wide angle video data without image data in (i.e.,without live image data or make-up image data) at the angle-of-view setat step S31. The condition of step S36 at FIG. 5 may correspond to thedarkened area of the panoramic image P10 illustrated in FIG. 1A.

Otherwise, if the controller 110 determines at step S34 that live imagedata exists for the angle-of-view set at step S31 at a time subsequentto the time corresponding to the present frame, the controller 110 atstep S35 sets the live image data recorded at the subsequent time as theimage data deployed in the angle-of-view set at step S31 for the presentframe. The process at step S35 may correspond to setting a make-up imagefor angle-of-views F12 and F13 in the example of FIG. 1B.

At step S37, the controller 110 determines whether the expansion ofimage data into all angles-of-view corresponding to the wide angle videodata has been completed. If it is determined at step S37 that the wideangle video data has been generated for all angles-of-view in the frame,the process of FIG. 5 is complete for the present frame and the processillustrated in FIG. 5 is repeated for each subsequent frame in the timeinterval corresponding to the wide angle video data. Otherwise, if thecontroller 110 determines at step S37 that the wide angle video data hasnot been expanded into all of the angles-of-view included in the wideangle video data frame, the controller 110 returns to step S31 andrepeats the process for a new angle-of-view that is not yet representedin the wide angle video frame.

Next, FIG. 6 illustrates an exemplary sequential flow diagram at a timeof recording video data, according to certain embodiments. The exampleof FIG. 6 refers to processing performed by a video manager 110 a, afile coupler 110 b, and an encoder 110 c, which may be included in thecontroller 110.

The sequence of FIG. 6 starts at step S101 where the controller 110outputs a recording start instruction to the video manager 110 a.

At step S102, the video manager 110 a confirms the current angle-of-viewand direction of the terminal device 100 with the sensor 108. Theconfirmation performed at step S102 may correspond to determining theorientation of the terminal device 100 at the time of recording thevideo data.

At step S103, the video manager 110 a provides the terminal device'sangle-of-view information to the file coupler 110 b.

At step S104, the video manager 110 a outputs an instruction to startrecording video image data to the file coupler 110 b, the encoder 110 c,and the camera 109. Recording processing is then performed at step S110,which includes the processing steps S111 through S118 illustrated inFIG. 6.

At step S111, the camera 109 captures image data and outputs the imagedata to the encoder 110 c.

At step S112, the encoder 110 c encodes the captured image data receivedfrom the camera 109.

At step S113, the encoded video stream is output to the file coupler 110b, and the video stream is stored by the file coupler 110 b in thememory 150.

At step S114, the file coupler 110 b stores the angle-of-viewinformation and frame number(s) associated with the video stream intoanother table that is different than the video stream.

If necessary, additional image data may be captured, encoded, and storedwith associated angle-of-view and frame number information in steps S115through S118. These steps may be repeated as many times as needed whenrecording the video data at a given angle-of-view.

Next, assume that the angle-of-view utilized when recording image datain the above steps changes at step 120. In response to the detectedchange of the angle-of-view, the processing of steps 121 through 128 isperformed.

At step S121, the sensor 108 detects the angle-of-view of the terminaldevice 100 following the change.

At step S122, the sensor 108 outputs a notification of the angle-of-viewchange to the video manager 110 a.

At step S123, the video manager 110 a transmits a notification signal tothe file coupler 110 b indicating the change in the angle-of-viewinformation.

At step S124, the video manager 110 a outputs an intra-frame request tothe encoder 110 c.

At step S125, the camera 109 captures image data at the newangle-of-view and transmits the captured image data to the encoder 110c.

At step S126, the encoder 110 c encodes the image data received from thecamera 109.

At step 127, the video stream encoded by the encoder 110 c istransmitted to the file coupler 110 b, and the file coupler 110 b storesthe encoded video stream in the memory 150.

At step S128, the file coupler 110 b stores the angle-of-viewinformation and frame number(s) associated with the encoded video streaminto a table that is different than the stored video stream.

At step S132, the controller 110 outputs an instruction to the videomanager 110 a instructing the video manager 110 a to stop recordingimage data.

At step S133, the video manager 110 a outputs an instruction to stoprecording image data the file coupler 110 b, the encoder 110 c, and thecamera 109.

At step S134, the encoder 110 c encodes the image data supplied to theencoder at step S131.

At step S135, the video stream encoded by the encoder 110 c is output tothe file coupler 110 b, and the file coupler 110 b stores the videostream in the memory 150.

At step S136, the file coupler 110 b stores angle-of-view informationand frame number(s) associated with the stored video stream in a tablethat is different than the stored video stream.

At step S137, the encoder 110 c outputs a notification to the filecoupler 110 b indicating a completion of encoding.

At step S138, the file coupler 110 b completes the storage of the videoimage data file generated in the previous steps.

Next, FIG. 7 illustrates an exemplary flow diagram of image dataprocessing at a time of video reproduction, according to certainembodiments. The exemplary flow diagram of FIG. 7 refers to processingperformed by the video manager 110 a, a file decoupler 110 d, and adecoder 110 e, which may be included in the controller 110.

At step S201, the controller 110 outputs an instruction to the videomanager 110 a indicating a start of image data reproduction.

At step S202, the video manager 110 a outputs an instruction to thesensor 108 requesting confirmation of the current angle-of-view anddirection of the terminal device 100. The information obtained at stepS202 may indicate the orientation of the terminal device 100 duringvideo reproduction.

At step S203, the video manager 110 a outputs an instruction to the filedecoupler 110 d to begin generating wide angle video data based on allframes of video data corresponding to all angles-of-view stored in thememory 150 for a given video sequence. In one or more embodiments, theprocessing performed at step S203 may correspond to the processingillustrated and described above with respect to FIG. 5.

At step S204, the decoder 110 e decodes all frames of video data of allangles-of-view represented in the wide angle video data.

At step S205, the controller 110 controls the display 120 such that thedisplay 120 displays all frames of the wide angle video data in a360-degree (or another angle corresponding to the total wide angle videodata field of view) view with the current viewing direction as thecenter of the view. The current viewing direction of the terminal device100 is determined based on the information obtained above at step S202.In one or more embodiments, the displayed wide angle video data fromstep S205 may be displayed with another position utilized as the centerof the displayed image. For example, the wide angle video data may bedisplayed such that the midpoint of the angle-of-view range of the wideangle video data is the center position when the video is displayed.

At step S206, the video manager 110 a outputs an instruction to the filedecoupler 110 d to start a decoding process for video reproduction.

At step S207, the video manager 110 a outputs an instruction to thedecoder 110 e to start a decoding process for video reproduction.

As video reproduction processing is performed in the terminal device 100using image data stored in the memory 150, the video reproductionprocessing continues at step S210, where the steps S211 through S214 areperformed repeatedly, as needed.

At step S211, the file decoupler 110 d reads out angle-of-viewinformation and frames corresponding to the angle-of-view received atstep S202.

At step S212, the file decoupler 110 d transfers the read-out data fromstep S211 to the decoder 110 e.

At step S213, the decoder 110 e decodes the transferred video stream.

At step S214, the decoder 110 e outputs the decoded video stream to thedisplay 120, where it is displayed. If necessary, steps S211 throughS214 may be repeated as steps S215 through S218 for subsequent videostreams at a given angle-of-view.

Next, it is assumed that the orientation of the terminal device changesat step S220 (i.e., the reproduction angle-of-view changes). Steps S221through S227 are performed in response to the change in the orientationof the terminal device 100.

At step S211, the sensor 108 detects an angle-of-view change of theterminal device 100.

At step S222, the sensor 108 outputs a notification of the angle-of-viewchange to the video manager 110 a.

At step S223, the video manager 110 a outputs a notification signal tothe display 120 notifying the display that the angle-of-view forreproduction on the display 120 has changed.

At step S224, the file decoupler 110 d reads out the frame(s)corresponding to the new angle-of-view, based on the notificationreceived at step S222.

At step S225, the file decoupler 110 d transfers the video streamcorresponding to the new angle-of-view to the decoder 110 e.

At step S226, the decoder 110 e decodes the received video stream. Atstep S227, the video stream decoded by the decoder 110 e is transferredto the display 120, and the display 120 displays the transferred videostream.

The steps S221-S227 may be repeated for subsequent video streamscorresponding to the new angle-of-view.

At step S231, the controller 110 outputs an instruction to the videomanager 110 a to stop reproducing image data. In response to receivingthe instruction to stop reproducing image data, the video manager 110 aoutputs an instruction to stop reproducing image data to the filedecoupler 110 d and the decoder 110 e at step S232.

Next, FIGS. 8A through 8C illustrate exemplary reproduction states ofpanoramic image data, according to certain embodiments. The panoramicimage data illustrated in FIGS. 8A through 8C includes a plurality offrames of image data that may correspond to a combination of live imagedata and make-up image data processed to form wide angle video data inaccordance with the present disclosure. For the purposes of the presentexample, the various frames illustrated in FIGS. 8A through 8C areassumed to be of a corresponding size to an angle-of-view at which thecamera 109 is capable of capturing image data. The panoramic image dataillustrated in FIGS. 8A through 8C is comprised of five frames; however,greater than or less than five frames may be utilized in otherimplementations of a device according to the present disclosure.

Referring first to FIG. 8A, panoramic image Pa includes image Pa-Ccorresponding to a frame at a center angle-of-view. Pa-R corresponds toan image frame at a right angle-of-view. Pa-U corresponds to an imageframe at an upper angle-of-view. Pa-L corresponds to an image frame at aleft angle-of-view. Pa-D corresponds to an image frame at a downwardangle-of-view.

The center angle-of-view and the surrounding angle-of-views are arrangedsuch that the image Pa is displayed in three dimensions in the exampleof FIGS. 8A-8C, and it is assumed that the display 120 of the terminaldevice 100 is capable of displaying image data in accordance with thisexample.

FIG. 8B illustrates one frame of image data at each of the angle-of-viewlocations included within the panoramic image Pa. In the example of FIG.8B, the image frame displayed in the position corresponding to thecenter angle-of-view is displayed at a different brightness level thanthe other displayed frames included in the panoramic image Pa. Forexample, the image displayed in the angle-of-view corresponding to Pa-Cmay be displayed at higher brightness than the surrounding images in theother angle-of-views. Other techniques may be implemented fordistinguishing one or more of the image frames in a panoramic image froma remaining set of image frames. For example, the image framecorresponding to Pa-C may be displayed normally while the surroundingimage frames are displayed in a semitransparent state.

FIG. 8C illustrates an example of delineating live image data within apanoramic image. In this example, the controller 110 controls thedisplay 120 such that a frame X is displayed within the panoramic imagePa, whereby the frame X indicates that the video image data displayedwithin the frame X is live image data. Accordingly, the display of liveimage data and make-up image data corresponding to the remaining portionof the panoramic image may be easily distinguished when viewing theimage Pa.

Next, FIGS. 9A and 9B illustrate exemplary relationships between arecording angle-of-view of a captured image and a reproductionangle-of-view of a displayed image, according to certain embodiments.

Referring first to FIG. 9A, FIG. 9A illustrates an example of anangle-of-view at a time of recording image data with the terminal device100, and a relationship between the angle-of-view at the time ofrecording and the image being captured. As shown in FIG. 9A, it isassumed that the image data in this example is captured by rotating theterminal device 100 by 180-degrees while recording the image data.Changes in live image data captured as the terminal device 100 isrotated in the example of FIG. 9A are shown above the illustration ofthe terminal device.

Referring now to FIG. 9B, FIG. 9B illustrates the angle (direction) ofthe terminal device 100 at the time of reproducing video image data, anda relationship with the image displayed at the various reproductionangles-of-view. It is assumed for the purposes of FIG. 9B that the videois reproduced on a display included in wearable glasses, such as thevideo reproduction apparatus 200 illustrated in FIG. 1C. At the time ofreproducing the video as shown in FIG. 9B, it is assumed that the videoreproduction apparatus 200 remains substantially fixed in a statewithout motion. Further, each image shown in FIG. 9A and each imageshown in FIG. 9B are images of the same timing for each of theillustrated rotation angles. Moreover, it is assumed that the size ofthe image reproduced with the eyeglasses-type video reproductionapparatus 200 is the same as that of one frame which was recorded withthe terminal device 100.

As shown in FIG. 9B, the reproduced image in each timing changes. Thatis, the video reproduction apparatus 200 only displays the live imagedata LV when the angle-of-view recorded with the terminal device 100corresponds with the reproduction angle-of-view of the videoreproduction apparatus 200. At the timings at which the angle-of-view ofreproduction and the angle-of-view at the time of recording the imagedata do not correspond, the video reproduction apparatus 200 displays acombination of live image data LV and make-up image data MV, or thevideo reproduction apparatus 200 may only display the make-up image dataMV. It should be noted that while the make-up image data MV in thisexample is illustrated as a darkened area when displayed on the videoreproduction apparatus 200, this is merely for illustration purposes,and the actual make-up image data may represent a scene captured at asubsequent time to the live image data LV.

Next, as mentioned above, wide angle video data may include acombination of live image data and make-up image data, whereby themake-up image data may be displayed as still image data. However, incertain implementations, the controller 110 may control the display 120such that make-up image data displayed within wide angle video data isdisplayed with simulated motion. For example, FIGS. 10A and 10Billustrate an example of simulating motion with a make-up imagedisplayed within wide angle video data, according to certainembodiments.

Referring first to FIG. 10A, FIG. 10A illustrates make-up image dataM11-M14 corresponding to a motor vehicle driving on a road within liveimage data LV. Specifically, make-up image data M11, M12, M13, and M14are displayed sequentially within frames of the live image data LV. Asillustrated in the figure, motion of the make-up image data may besimulated by changing one or more of a display position or display sizeof the moving image data within the live image data.

For example, as shown in FIG. 10B, there may be a case in which motorvehicle image M11′ is included within make-up image MV. In certainimplementations, the controller 110 may control the display such thatthe make-up image MV (or M11′) is reduced in size gradually over time,resulting in motor vehicle images M12′, M13′, and M14′. Thus, when themotor vehicle images in FIG. 10B are viewed in sequential frames, thedecreasing/increasing size of the motor vehicle image simulates motionof the motor vehicle on the road. Additionally, in certain embodiments,when motor vehicle images M12′, M13′, and M14′, which are reducedversions of image M11′, are combined with the make-up image MV, anon-display section Y is generated based on the difference between thesizes of the images. In certain embodiments, this non-display section Ymay be displayed in black, or may be displayed in another color such asthe color of a road, the color of a sky, etc.

Next, FIGS. 11A through 11C illustrate another non-limiting example ofsimulating motion of a make-up image within wide angle video data,according to certain embodiments. Referring first to FIG. 11A, FIG. 11Aillustrates an example in which make-up image data corresponding to ahuman figure is displayed sequentially within live image data LV.Specifically, make-up images M21, M22, M23, and M24 illustrate a humanfigure that appears to be in motion with respect to the backdrop of liveimage data LV. For example, as shown in FIG. 11B, a motion of the humanfigure in make-up images M21 through M24 may be simulated in a walkingmotion by changing a display size and/or display position of the figurein a repeating pattern corresponding to patterns PT1 and PT2. Further,as illustrated in FIG. 11B, aspects of the make-up images may be variedin their entirety, or a portion thereof may be changed in size/positionto simulate the motion. In the example shown in FIG. 11C, human figuresrepresenting make-up images M25, M26, M27, and M28 within make-up imageMV are shown, whereby the images M25 and M27 correspond to the shapes ofpattern PT2 and the images M26 and M28 correspond to the pattern PT1. Byvarying the make-up image data as illustrated in FIGS. 11A through 11C,the display 120 may output make-up image data including images of humanfigures such as images M25 through M28 such that variations in themake-up image data appear as simulated motion within other image data.

Next, FIG. 12 illustrates a non-limiting exemplary sequential flowdiagram for simulating motion with a make-up image, according to certainembodiments. The sequential flow diagram of FIG. 12 may correspond tothe simulated motion illustrated and discussed above with respect toFIGS. 10A-11B. The flow diagram of FIG. 12 includes steps discussedabove with respect to FIG. 6. Accordingly, for the sake of brevity, adescription of these steps will not be repeated here. Steps S141 throughS144 are added in FIG. 12. Moreover, FIG. 12 illustrates processing withrespect to an object tracker 110 f, which may be included in thecontroller 110. The object tracker 110 f may perform image analysis suchthat an object within an image is detected and extracted. Exemplaryobjects that may be detected within an image include a human figure, amotor vehicle, or another arbitrary object.

At step S141, an image is captured with the camera 109 and the capturedimage is transferred to the object tracker 110 f.

At step S142, the object tracker 110 f analyzes the image and detects anobject within the image.

At step S143, the object tracker 110 f transfers the pattern of thedetected object from the image to the file coupler 110 b.

At step S144, the file coupler 110 b stores the pattern of the detectedobject in the memory 150.

Next, FIG. 13 illustrates a non-limiting exemplary sequential flowdiagram for reproducing images with simulated motion from a make-upimage, according to certain embodiments. The sequential flow diagram ofFIG. 13 may correspond to the simulated motion illustrated and discussedabove with respect to FIGS. 10A-11B. The exemplary flow diagram of FIG.13 includes steps that were illustrated and discussed above with respectto FIG. 7. Accordingly, the repeated steps will not be described herefor the sake of brevity. Steps S241 through S244 are added from FIG. 7.Moreover, the flow diagram from FIG. 13 describes processing withrespect to a pattern reproducer 110 g, which may be included in thecontroller 110. Using patterns stored in the memory 150 (as discussedabove with respect to FIG. 12), the pattern reproducer 110 g maygenerate a pattern (object) for reproduction by combining the patternwith another image during the reproduction.

At step S241, the file decoupler 110 d reads the pattern saved in thememory 150.

At step S242, the file decoupler 110 d transfers the read pattern to thepattern reproducer 110 g.

At step S243, the pattern reproducer 110 g reproduces the patterncombined in another image, such as live image data or another make-upimage.

At step S244, the pattern reproducer 110 g outputs the reproducedpattern combined with the other image data to the display 120, and thedisplay 120 outputs the image with the reproduced pattern.

Next, FIG. 14 illustrates a non-limiting example of capturing image datawith cameras mounted on multiple sides of a terminal device, accordingto certain embodiments. In this example, it is assumed that a frontfacing camera 109F is mounted on a front surface of the terminal device100 and a rear facing camera 109R is mounted on a rear surface of theterminal device 100. Accordingly, by providing cameras mounted on theterminal device 100 as shown in FIG. 14, when capturing panoramic imagedata around the terminal device 100, the live image data LV may beobtained with respect to two angles-of-view for each timing at which theimage data is captured. Accordingly, when performing processingdescribed herein and generating wide angle video data in accordance withthe present disclosure, the angle-of-view represented by the live imagedata LV may increase, and as a result, the amount of the make-up imagedata MV used to generate the wide angle video data may decrease.

For example, as shown in FIG. 14, it is assumed that the totalangle-of-view of wide angle video data generated with the terminaldevice 100 is 360-degrees, comprising frames from a total of sixangles-of-view. By capturing image data from multiple angles as in FIG.14, live image data LV may be captured from two of the sixangles-of-view. The four remaining angles-of-view may be comprised ofmake-up image data MV.

Obviously, numerous modifications and variations of the presentdisclosure are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as specifically describedherein. For example, advantageous results may be achieved if the stepsof the disclosed techniques were performed in a different sequence, ifcomponents in the disclosed systems were combined in a different manner,or if the components were replaced or supplemented by other components.The functions, processes and algorithms described herein may beperformed in hardware or software executed by hardware, includingcomputer processors and/or programmable processing circuits configuredto execute program code and/or computer instructions to execute thefunctions, processes and algorithms described herein. A processingcircuit includes a programmed processor, as a processor includescircuitry. A processing circuit also includes devices such as anapplication specific integrated circuit (ASIC) and conventional circuitcomponents arranged to perform the recited functions.

Further, the generation of wide angle video data such as the processshown in FIG. 5 may be performed at a time of recording image datarather than a time of reproducing image data for display. In one or moreembodiments, a terminal device may be made to perform a wide angle videodata generation process at a time other than recording or reproduction.

The functions and features described herein may also be executed byvarious distributed components of a system. For example, one or moreprocessors may execute these system functions, wherein the processorsare distributed across multiple components communicating in a network.The distributed components may include one or more client and/or servermachines, in addition to various human interface and/or communicationdevices (e.g., display monitors, smart phones, tablets, personal digitalassistants (PDAs)). The network may be a private network, such as a LANor WAN, or may be a public network, such as the Internet. Input to thesystem may be received via direct user input and/or received remotelyeither in real-time or as a batch process.

Moreover, the reproduced images in FIG. 8 and the other figures aremerely examples, and the reproduction of image data may be performedsuch that the display outputs images differently than these examples.

Additionally, some implementations may be performed on modules orhardware not identical to those described. Accordingly, otherimplementations are within the scope that may be claimed.

It must be noted that, as used in the specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise.

The above disclosure also encompasses the embodiments noted below.

(1) A device comprising: one or more image sensors that capture livevideo data; a memory that stores the captured live video data; circuitryconfigured to determine an angle-of-view for one or more frames of thelive video data; generate wide angle video data that has a largerangle-of-view than the one or more frames of the live video data;determine one or more angles-of-view not represented in the wide anglevideo data by the live video data; generate make-up image data capturedat a later time than the live video data, wherein the make-up image datahas an angle-of-view corresponding to at least one of the one or moreangles-of-view that are not represented in the wide angle video data bythe live video data; and update the wide angle video data to include themake-up image data.

(2) The device of (1), wherein the one or more image sensors areconfigured to generate a frame number for each frame of the live videodata and to store the frame number with the live video data in thememory.

(3) The device of (1) or (2), wherein the circuitry is furtherconfigured to generate the wide angle video data by including themake-up image data at a corresponding frame number of the live imagedata.

(4) The device of any one of (1) to (3), wherein the circuitry isfurther configured to continuously include the make-up image data in thewide angle video data until a time corresponding to when the make-upimage data was captured live.

(5) The device of any one of (1) to (4), further comprising a display,wherein the circuitry is further configured to control the display suchthat the wide angle video data is displayed on the display.

(6) The device of any one of (1) to (5), wherein the circuitry isfurther configured to control the display such that when the wide anglevideo data is displayed, a portion of the wide angle video is displayeddifferently than a remaining portion of the wide angle video data.

(7) The device of any one of (1) to (6), further comprising one or moresensors configured to determine an orientation of the device, whereinthe circuitry is further configured to control the display, based on thedetermined orientation of the device, such that the portion of the wideangle video data that is displayed differently has an angle-of-viewcorresponding to the orientation of the device.

(8) The device of any one of (1) to (7), wherein the circuitry isfurther configured to control the display such that the portion of thewide angle video data is displayed with a different brightness levelthan the remaining portion of the wide angle video data.

(9) The device of any one of (1) to (8), further comprising one or moresensors configured to determine an orientation of the device, whereinthe circuitry is further configured to control the display, based on thedetermined orientation of the device, such that a portion of the wideangle video data that corresponds to the live video data is delineatedon the display.

(10) The device of any one of (1) to (9), wherein the portion of thewide angle video data that corresponds to the live video data changes asthe orientation of the device changes.

(11) The device of any one of (1) to (10), wherein the circuitry isfurther configured to control the display such that when the orientationof the device corresponds to an angle-of-view in the wide angle videodata that is not represented by the live video data, make-up image datais displayed in the angle-of-view that is not represented by the livevideo data.

(12) The device of any one of (1) to (11), wherein the display isimplemented in wearable glasses.

(13) The device of any one of (1) to (12), wherein the circuitry isfurther configured to generate the wide angle video data by varying, forsequential frames of the wide angle video data, one or more of a sizeand an orientation of a portion of the displayed make-up image data.

(14) The device of any one of (1) to (13), wherein the portion of themake-up image data is varied such that the portion appears to be movingwhen the wide angle video data is displayed.

(15) The device of any one of (1) to (14), wherein the circuitry isfurther configured to detect when the portion of the make-up image datacorresponds to a human figure.

(16) The device of any one of (1) to (15), wherein the circuitry isfurther configured to repetitively vary the one or more of the size andthe orientation of the portion of the make-up image data when it isdetermined that the portion corresponds to the human figure.

(17) The device of any one of (1) to (16), wherein: the device includesat least two image sensors, and the at least two image sensors areconfigured to simultaneously capture the live video data from theirrespective angles-of-view.

(18) The device of any one of (1) to (17), wherein at least one pair ofthe at least two image sensors are oriented on opposing sides of thedevice.

(19) A method comprising: receiving, from one or more image sensors,live video data captured by the one or more image sensors; storing thelive video data in a memory; determining, by circuitry, an angle-of-viewfor one or more frames of the live video data; generating, by thecircuitry, wide angle video data that has a larger angle-of-view thanthe one or more frames of the live video data; determining, by thecircuitry, one or more angles-of-view not represented in the wide anglevideo data by the live video data; generating, by the circuitry, make-upimage data captured at a later time than the live video data, whereinthe make-up image data has an angle-of-view corresponding to at leastone of the one or more angles-of-view that are not represented in thewide angle video data by the live video data; and updating, by thecircuitry, the wide angle video data to include the make-up image data.

(20) A non-transitory computer readable medium having instructionsstored therein that when executed by one or more processors cause adevice to perform a method comprising: receiving, from one or more imagesensors, live video data captured by the one or more image sensors;storing the live video data in a memory; determining an angle-of-viewfor one or more frames of the live video data; generating wide anglevideo data that has a larger angle-of-view than the one or more framesof the live video data; determining one or more angles-of-view notrepresented in the wide angle video data by the live video data;generating make-up image data captured at a later time than the livevideo data, wherein the make-up image data has an angle-of-viewcorresponding to at least one of the one or more angles-of-view that arenot represented in the wide angle video data by the live video data; andupdating the wide angle video data to include the make-up image data.

The invention claimed is:
 1. A device comprising: a memory that storesvideo data; circuitry configured to determine an angle-of-view for oneor more frames of the video data; generate wide angle video data thathas a larger angle-of-view than the one or more frames of the videodata; determine one or more angles-of-view not represented in the wideangle video data by the video data; generate make-up image data based ona portion of the video data, wherein the make-up image data has anangle-of-view corresponding to at least one of the one or moreangles-of-view that are not represented in the wide angle video data bythe video data; and update the wide angle video data to include themake-up image data.
 2. The device of claim 1, wherein a frame number isgenerated for each frame of the video data and stored with the videodata in the memory.
 3. The device of claim 2, wherein the circuitry isfurther configured to generate the wide angle video data by includingthe make-up image data at a corresponding frame number of the videodata.
 4. The device of claim 1, wherein the circuitry is furtherconfigured to continuously include the make-up image data in the wideangle video data until a time corresponding to when the video data, uponwhich the make-up image data is based, was captured.
 5. The device ofclaim 1, further comprising a display, wherein the circuitry is furtherconfigured to control the display such that the wide angle video data isdisplayed on the display.
 6. The device of claim 5, wherein thecircuitry is further configured to control the display such that whenthe wide angle video data is displayed, a portion of the wide anglevideo data is displayed differently than a remaining portion of the wideangle video data.
 7. The device of claim 6, further comprising one ormore sensors configured to determine an orientation of the device,wherein the circuitry is further configured to control the display,based on the determined orientation of the device, such that the portionof the wide angle video data that is displayed differently has anangle-of-view corresponding to the orientation of the device.
 8. Thedevice of claim 6, wherein the circuitry is further configured tocontrol the display such that the portion of the wide angle video datais displayed with a different brightness level than the remainingportion of the wide angle video data.
 9. The device of claim 5, furthercomprising one or more sensors configured to determine an orientation ofthe device, wherein the circuitry is further configured to control thedisplay, based on the determined orientation of the device, such that aportion of the wide angle video data that corresponds to the video datais delineated on the display.
 10. The device of claim 9, wherein theportion of the wide angle video data that corresponds to the video datachanges as the orientation of the device changes.
 11. The device ofclaim 10, wherein the circuitry is further configured to control thedisplay such that when the orientation of the device corresponds to anangle-of-view in the wide angle video data that is not represented bythe video data, make-up image data is displayed in the angle-of-viewthat is not represented by the video data.
 12. The device of claim 5,wherein the display is implemented in wearable glasses.
 13. The deviceof claim 1, wherein the circuitry is further configured to generate thewide angle video data by varying, for sequential frames of the wideangle video data, one or more of a size and an orientation of a portionof the displayed make-up image data.
 14. The device of claim 13, whereinthe portion of the make-up image data is varied such that the portionappears to be moving when the wide angle video data is displayed. 15.The device of claim 13, wherein the circuitry is further configured todetect when the portion of the make-up image data corresponds to a humanfigure.
 16. The device of claim 15, wherein the circuitry is furtherconfigured to repetitively vary the one or more of the size and theorientation of the portion of the make-up image data when it isdetermined that the portion corresponds to the human figure.
 17. Thedevice of claim 1, wherein: the device includes at least two imagesensors, and the at least two image sensors are configured tosimultaneously capture the video data from their respectiveangles-of-view.
 18. The device of claim 17, wherein at least one pair ofthe at least two image sensors are oriented on opposing sides of thedevice.
 19. A method, implemented by a device, comprising: storing, at amemory, video data; determining, by circuitry of the device, anangle-of-view for one or more frames of the video data; generating, bythe circuitry, wide angle video data that has a larger angle-of-viewthan the one or more frames of the video data; determining, by thecircuitry, one or more angles-of-view not represented in the wide anglevideo data by the video data; generating, by the circuitry, make-upimage data based on a portion of the video data, wherein the make-upimage data has an angle-of-view corresponding to at least one of the oneor more angles-of-view that are not represented in the wide angle videodata by the video data; and updating, by the circuitry, the wide anglevideo data to include the make-up image data.
 20. A non-transitorycomputer readable medium having instructions stored therein that whenexecuted by one or more processors cause a device to perform a methodcomprising: storing, at a memory, video data; determining anangle-of-view for one or more frames of the video data; generating wideangle video data that has a larger angle-of-view than the one or moreframes of the video data; determining one or more angles-of-view notrepresented in the wide angle video data by the video data; generatingmake-up image data based on a portion of the video data, wherein themake-up image data has an angle-of-view corresponding to at least one ofthe one or more angles-of-view that are not represented in the wideangle video data by the video data; and updating the wide angle videodata to include the make-up image data.